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ABSTRACT 


Sources of Values Influencing Educational 
Evaluation ; 


Nick L. Smith 


Northwest Regional Educational Laboratory 


Social and personal values from a multitude of sources influence the 
conduct of evaluation studies in education. This paper discusses the 
impact of two major sources of such values: (a) contextual. factors in 
evaluation studies, including the political role of evaluations, 
current Soeiail movements and organizational influences, and (b) evalua- 
tors themselves, including evaluators' terminology, models, and personal 
values. The influences of values on the purposes of evaluation studies 
and ‘on evaluator roles are discussed as illustrative examples. Finally, 


four’ means of clarifying values in evaluation work are identified. 
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SOURCES OF VALUES INFLUENCING EDUCATIONAL EVALUATION 


We may thus expect that while personality factors 
will give rise to variations in individual value 
systems, cultural, institutional, and social 
factors will nevertheless restrict such variation 
to a reasonably small number, perhaps a few 
million. (Rokeach, 1968-1969, p. 552) 


Myriad value judgments of what is good or ideal arise from these 
millions of value systems to influence the design and conduct of every 
evaluation study in education. Social movements, political pressure 
groups, organizational influences and standard evaluation techniques 
are some of the sources of these value systems which support the 
pervasive but largely unrecognized, .unchallenged judgments of the 
quality of evaluation. Improving educational evaluation as a form of 
both scientific inquiry and social service requires a better under- 
standing of how these implicit judgments shape evaluative activities. . 

As practitioners, evaluators are often preoccupied with making 
value judgments about an educational program's effectiveness, while 
seldom conscious of the more subtle value judgments which launch and 
guide the evaluation study itself. But it is sometimes important for 
funding agencies, clients, audiences, and especially the evaluators | 
themselves to be aware of these latter judgments. Such groups need to 
know what was left out of an evaluation study, and why, as well as 
what was included in it. As theorists, evaluators need to understand 
how evaluation studies are part of a larger socio-political enterprise 
of competing value systems. As an initial step in understanding the 
role of values in evaluation, I will highlight in the following pages 
the major sources of values which influence how and why evaluation 
studies are conducted. 

Discussing the role. of values in evaluation is best done from as 
neutral a stance as possible towards evaluation itself. As argued by 
Scriven (1972), it seems most beneficial to define evaluation as 
broadly and néutrally as possible without the definition implying in 
what ways evaluation is appropriate/inappropriate, useful/not useful, 
desirable/undesirable, or ethical/unethical. Such judgments should be 


based on a consideration of relevant value positions in specific 


“J 


7 evaluation settings. Since it is misleading to group these evaluative 
concerns as part of the definition of evaluation, evaluation is defined 
here as simply the assessment of worth. Subsequent discussions of 
competing value systems will illustrate how the same study may be 
considered exemplary, trivial, or unethical, depending on one's value 


perspective. 


THE NATURE OF VALUES IN EVALUATION WORK: 
“& 
Value Definitions 
Although evaluation by definition deals with values, the 

literature of educational evaluation contains little treatment of how 
one identifies, conceptualizes, or operationalizes values in evaluation 
work. In fact, the term "values" remains vague and ambiguous. Berlak 
(1970), for example, defines value as a 


belief or conjunction of beliefs which guide human 
behavior. Moral values differ in that they are 
beliefs that establish ideals or standards for 
action. Thus vigilantism has been called a common 
value to most Americans...but the latter is not a 
moral value. Equality, honesty, and human dignity 
would be classified as moral values. (p. 266) 


Hodgkinson (1970), in studying organizational influences on value 
systems, defines value as "a conception of the desirable" (p. 46). He 
distinguishes between values and attitudes which are defined as 
"predispositions to act towards referents in a consistent manner. 
Attitudes then are the manifest or overt indicants of value" (ee 46) . 


Rokeach (1968-1969), a senior researcher in the field of values and 


attitudes, states that 


there is still very little consensus about the ee 
exact conceptual difference between an attitude 

and a value; we sometimes employ these two 

concepts interchangeably and sometimes 

differentially. We sometimes employ them in the 
singular and sometimes in the plural, as if we 

have not yet learned how to count them. (pp. 549-550) 
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Finally, Goulet (1971) has succinctly summarized the problem as 
follows: , 


in. common discourse, "values" refer in a general 
"way to attitudes, preferences, life-styles, 
normative frameworks, symbolic universes, belief 
systems, and networks of ‘meaning, men give to life. 
Sociologists, psychologists, philosophers, and 
others have always had great difficulty in 
defining the term with precision. (p. 205) 


Scriven (1967) has assisted evaluators with this definitional 
problem; he discusses three types of value judgments, arguing that they 
have a preper place in evaluation, cannot be ALaregasded, and should 
fot be used to demean evaluation as non-scientific. The first type of. 
value judgments are simply statements about personal preferences or 
matters of taste. These statements can be established or refuted by 
showing whether or not someone holds them. The second type of value .* 


judgments are those related to the assessment of comparative merit or 


~ worth. Such judgments assess whether or not individuals hold certain 


comparative statements to be true and whether or not the statements are, 
in-.fact, true; e.g., whether or not one product is better than another. 
These value judgments Scriven describes as being complex combinations of 
various performance ratings and wedahtings of performance ‘criteria. 

This is the kind of judgment most frequently employed by evaluators to 
assess a program's effectiveness. 

The third type of value Juidieniene, and those of major concern here, 
are those in which the criteria themselves are debatable or wherein the 
question of what is good is at issue. These judgments are not considera- 
tions of what is, in fact, actual, but of what is to be donsidered good 
or ideal. Much of the form and faaws of current eval naeion work is 
determined by implicit value judgments of this type. Upon inspection, 
one can identify gallus poatbisns inherent in both general evaluation 
‘methodology and in the contexts of specific studies which dictate 


which evaluation activities are to be considered desirable. 


Situational Nature of Values \ 


° 


Although there are broad dominant values which affect all 


“evaluation studies, the influence of many values is situation specific. 


Worthen (1972) has noted that , 


cS 


the various [evaluation] models are built on 
differing--often conflicting--conceptions and 
definitions of evaluation, with the end. 
result that practitioners are®led in very 
different directions, depending upon which 
model they follow. (p. 3) 


I believe the divergence of these evaluation models reflects the highly 
contextual nature of evaluation work and mepules from models being 
developed for differing purposes and contexts. Focusing attention more 
directly on local values and conditions should therefore facilitate 
understanding how evaluation studies are conducted. , 

Gowin (1973) has suggested that whatever regularities researchers 
find in educational phenomena are there as a result of human beings 
acting within a social context. More traditional approaches to 
scientific investigation, such as that discussed by Platt (1964), 
presuppose thar the phenomena being studied have a natural stability. - 
Gowin argues that such approaches to science serve as poor models for 
educational research because the phenomena of interest in education are 
dynamic rather than stable. Educational phenomena are produced by 
deliberate choice and are mutable, not immutable. He argues that regu- 
larities do exist in educational phenomena, but calls them artifactual 
to stress that the regularities are man-made, and not due to underlying 
laws of nature. Gowin maintains that educational research should be 
concerned with isolating the empirical consistencies found in what are 
actually artifactual regularities arising in education from the 
interaction of human beings in specific social-contexts. Educational 
research should not be concerned with the search for underlying 
immutable laws of nature. 

Cronbach (1975) has made a similar argument about psychological 
research. He argues that research in psychology should attempt’ to 
describe and evaluate a given problem afresh in each new setting, 
attending heavily to local characteristics. One should concentrate 
on local conditions, not generalization. 


The goal of our work, I have argued here, is-.not 
to amass generalizations atop which a theoretical 
tower can some day be erected...The special task 
of.the social scientist in each generation is to 
pin down the contemporary facts. Beyond that, 

he shares with the humanistic scholar and the 
artist in the effort to gain insight into 
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contemporary relationships, and to realign the 

F culture's view of man with present realities. 
To. know man as he is, 1S no mean aspiration. 
(Cronbach, 1975, p. 126) 


Such an admonition applies equally well in evaluative research. 
Understanding why evaluations are conducted as they are requires 
greater sensitivity to those’ specific values which influence the 
formation of evaluation studies in ‘digiaual sekbings The first task 
then is to identify the sources of dominant values in various evaluation 
settings. In the following pages I identify several major sources of 
values and discuss their general influence on the nature of evaluation 


work in education.~ 


SOURCES OF VALUES IN AN EVALUATION'S CONTEXT 


Several sources of values which influence an evaluation study can 
be identified upon examining an evaluation's context or setting. These 
sources are aspects of the socio-political milieu within which the 
evaluation is conducted. Such sources include relevant political 


groups, current social movements, and organizational influences. 


Evaluation Contexts 

Gowin and Millman (1969) have defined a "context of inquiry" to 
include "the scene of phenomena of interest, telling questions and 
principles of evidence, key concepts and conceptual systems, basic 
assumptions and presuppositions" (p. 554) . They suggest that analyzing 
the scene of any research area will reveal pervasive qualities which 
will help determine the direction of the research. When the elements 
of the context of inquiry are analyzed, it is possible to expose the 
basic assumptions and presuppositions of the inquiry. 

In evaluation, House (1973a) has discussed the role of values in 
three types of context in formal evaluation: the context of evaluation, 
the context of justification, and the context of persuasion. The 
context of evaluation, according to House, refers to the basic value 
positions responsible for the genesis of the evaluation and includes 
all the motivations, biases, attitudes, and pressures from which the 


evaluation arose. 
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The context of justification is reflected in the attempts made to 


justify findings. While there are many ways of justifying findings, in 
formal educational evaluation such justification usually involves the 
methodology of the social sciences, principally psychology, in collect- 
ing mith analyzing data. House regards most evaluations as operating 
within the context of justification. 

House also discusses the context of persuasion, noting that 
producing data is: one thing and getting it used is quite another. 
Because of the political process within which evaluation usually 
operates, the context of, justification (i.e., the usé of social science 
values and methods) does not insure that evaluation results have social 
utility. Factors related to the context of persuasion largely deter- 
mine whether or not evaluation results get used. Although. evaluators 
frequently neglect the context of persuasion, they must assume some 
responsibility for their information being properly understood and used. 

I define the context of an evaluation study to include the severe 
classes of situational influences or conditions relevant to a full \ 
description of the environment or setting of the evaluation study. \guch 
conditions or influences certainly vary across evaluation studies but, 
include both broad factors, such as: ; 


social convention, 

historical background and tradition, 

economic climate, 

legal precedents, 

current political issues, 

dominant religious and philosophic orientations, 
social movements, 


and more specific factors, such as: 


control of evaluation resources, 

authority and decision processes, 

funding directives and contractual obligations, 
expectations about evaluation activities and outcomes, 
timing of evaluation work, and 

desires of client groups. 


The explicit and implicit values arising from these sources result in 
value judgments by funders, clients, and evaluators as to what ig the 
desirable or proper form of an evaluation study. \ 

Such factors exist in all evaluation studies, but the relative 
impact of any specific factor must be determined anew in each evj\luation 


setting. While social convention or control of resources may ai het an 
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overriding influence on the form of some evaluation studies, they may. 
have negligible impact in other evaluation settings. Which-values most 
influence evaluation studies vary substantially from setting to setting. 
‘Keeping in mind the situational nature of such value influences, 
let us now consider in turn three primary sources of values within an 


evaluation's context: the political aspects of evaluation work, the 


influence of*social movements, and organizational influences. 


Political Aspects of Evaluation 
Cohen (1970) has warned that lack of attention to evaluation context 
can result in biased and irrelevant criteria, questions, id data. He i 
notes that the evaluation of social action programs is only secondarily 
a scientific endeavor; "first and foremost it is an effort to gain 
, politically significant information on the consequences of political 
acts" (pp. 236-237). 
The conflicts arising from different value bases are frequently 
described as the "political" aspects of evaluation. Considerable ; 
writing has been done on pclitical settings and political influences 
on evaluation work. House (1973a), for example, talks about the polliti- 
cal pressures (value interests) which initiate most evaluation studies. 
My point is that evaluations are not inspired by \ 
the heartfelt need of professionals to try to do 
a better job (frequent though that rhetoric is) ; 


rather, the impetus is usually traceable toa \ 
pressure group with a specific aim. (p. 128) 


In discussing the differing political value bases which underlie 
the initiation of evaluation studies and influence evaluation activi- 
ties, House identifies the use of evaluations for attack, for defense, 
and evaluations initiated as a matter of form--symbolic gestures that 
are interpreted as such by the recipient of the, funds. He notes that 
the, primary problem with all these evaluations is not ther lack of 
rigorous, scientific methodology, byt the fact that they were never 
intended to produce relevant information. Evaluations for attack 
are generally more carefully done, he says, because they are trying 


to change things,rather than to defend an established position. o 


Weiss (1973), who is also concerned with the political nature of 
evaluation, cites three ways in which evaluation work and politics , - 
interminglé: 


policies concerning the programs being evaluated | 
are proposed, debated, enacted, funded, and Zz 
implemented through political processes, 


evaluation is frequently undertaken in order to 
assist decision-making and, therefore, its 
reports enter the political.arena, 


evaluation, by its own nature, has a political 
stance in terms of what it legitimizes and the 
evaluator's appropriate role in policy and 
program formation. 


In Weiss' opinion, only the evaluatar is sensitive to the 
political elements of the evaluation situation can.be as creative and 
strategically useful as he or she should be. She notes that because 
of the political context within*which programs originate and operate, 
political considerations can daily shape the work of the program, and, 
therefore, shape the concept of what it means $e evades the program. 
Values which etd evaluation activities originate in such political 
sources as ideological preferences, congressional support, public 
appraisal, competitors, client dissatisfaction, and media coverage. 
Since evaluation activities may serve decision making, they enter the 
political arena, can be used for political purposes, and are subject 
to political pressures. | 

Weiss mentions that one way political values influence evaluation 
studies is by limiting investigation to the effects of the experimental 
variables which the program manipulates. Such limitation conveys the 
message that other facets of the program are either unimportant or 
fixed and unchangeable. Such exclusive emphasis on a program's inter- 
vention components leads to a disregard for other important components, 
such as the social and institutional structures within which the 
program operates and the values of potential target groups. Thus 
politically relevant values can take precedence over other value 


orientations. 
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Social Movements ~ xf 

Social movements in education, such as the accountability movement, 
systems management movement, and the behavioral objectives movement are 
another sigtificant source of values which determine how evaluations are 
conducted. These movements are broad, essentialjy value-based, 
Orientations that prescribe "proper" evaluation approaches. 

House (1973b) has attacked the primary value base of the 
. accountability movement: economic rationality.- "For the dominant theme 
today is economic efficiency and its purpose is control--control over 
the behavior of pupils, control over the behavior of the staff, control 
over the schooling" (p. 261). House suggests that this value 
orientation presumes that the purpose of education is to supply manpower 
to other institutions in apeiety, tha# educational goals are set 
primarily by technological demands, and that these goals should be met 
by educators with the greatest efficiency possible. He argues that 
such educational goals are ultimately economic and the attendant 
accountability is also economic. The principles and values of economic 
efficiency thus ae as both the accountability system and the nature of 
education. ; 

House contrdsts this economic orientation with a more humanistic ae 
education, in which schools are assessed in terms of what they do for 
penile, independent of their contribution to other social institutions. 
He critiques the basic guiding principles of the accountability movement 
showing that they are simply ideological value assumptions, not the ? 
outgrowth of empirical or historical research. While the accountability 
movement may essentially be a value-based endeavor not supported by 
scientific evidence’ or methodology, there is little doubt that the 
movement's attendant values of prespecification, replication, quantifi- 
cation, and elimination of ambiguity have had a substantial impact on 
present evaluation methods. 

Likewise, systems management procedures, in many ways similar to 
some evaluation models, have been attacked because of their basic value 
positions. tn discussing the values evident in, and perpetuated 
through, such supposedly scientific procedures as systems management, 
Apple (1972) notes how the "systems" language performs a political 


function by convincing others of the sophisticated state of the field 
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of education and by evoking tacit meanings from the geneyal audience 
that are basically supportive of a quasi-scientific beldet system. He 
points out that order and consensus are extremely important for proper 
functioning of systems management procedures; confligt and disorder are 
antithetical to a smoothly functioning system. Buy, "the fact that 
conflict and disorder are extraordinarily importapit to prevent the 
reification of institutional patterns of action 


(p. 14). 


is, thus, ignored" 


It should: be clear, then, that sys 
are not essentially neutral, nor e they only 
performing a "Scientific" functign. By tending 
to cause its users and the othey publics involved 
to ignore certain possible fundamental problems 
with schools as institutions, /systems management 
also acts to generate and channel political 
sentiments supportive of th¢ existing modes,of 
access to knowledge and power. (p. 15) 


ms approaches 


Apple further points out that behavioral objectives and similar 
behavioristic rationales are genefaljy treated as if they were logically 
founded and scientifically arguable, instead of what they really are: 
expressions of a dominant, industrialized consciousness that seeks 
certainty above all else. 


That is, they are social and ideological configura- 
tions stemming from and mirroring a set of basic 
rules of thought that are part of the taken-for- 
granted reality of curriculum workers and other 
educators. The reality inclines us to search for 
relatively easy ways to eliminate the human 
dilemmas (even mysteries) of dealing with diversity 
and alternative conceptions of valued activity. 

(p. 17) 


Apple suggests that systems management procedures actually enable 
institutional managers to avoid conflicts over basic values by making 
‘choices within the limited framework of existing modes of interaction 
and by precluding questions about the basis of the structure itself. 
Using scientific terminology this way serves to de-emphasize value 
conflicts and to portray decisions as factual or reality-based 
judgments. 

Social movements, then, such as the behavioral objectives and 
accountability mevemenks are sources of values which foster hospitdble 


climates for only certain kinds of evaluation methodologies. The 
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expectations arising from such social values clearly constrain the 


evaluator's choice of evaluative techniques. 


Organizational Influences 


Just as broad social values shape individual evaluation studies, so 
specific organizational values may influence evaluations conducted 
within an organization. Walker (1974) discusses how the role of 
evaluation in educational research and development agencies ‘is 
influenced by alternative structural, organizational, and managerial 
options. Different types of organizations consider different forms of 


methodology more or less appropriate. Schmidtlein (1974) lists several 


“value orientation dilemmas" which he uses to describe the value 
orientations of common organizational decision-making processes. One 
can expect these organizational values to be represented in the evalya- 
tion work of organizational members. 

Indeed, there is empirical evidence to suggest a relationship 
between organizational values and the values held by individual staff 
fenbere. In studying chemists, Hinrichs (1972) found that eight years 
after attaining their doctorates, chemists who worked in industrial 
research differed significantly in their profession-oriented values 
from chemists who worked in academic settings, even when the two groups 
were equated in terms of their prevailing values upon attaining their 

_Soctorates. Hinrichs draws the implication that differential changes 
occurred in these two groups as a result of their exposure to separate 
environments. He notes that even chemists who changed settings (that 
is, went from an academic to an industrial environment or vice versa) 
changed some of their values in order to be more consistent with the 
average value patterns of their new colleagues. 


This study has illustrated that the phenomenon of 
accommodation or acculturation in values does 
exist during the course of a scientist's early 
career, and that such acculturation is to some 
extent dependent upon the working environment 
into which the scientist enters. (p. 563) 


In an empirical investigation of the relationship between values 
of educational personnel and their position in organizational hierarchies, 
Hodgkinson (1970) found that yalues and organizational position are 
Clearly related. Evidence suggests that an individual's values change 
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with progression through the ranks of an organizational hierarchy. As 
Hodgkinson notes, it is not clear whether personnel change their values 
because of promotions, or that those personnel with what he calls 
"flexibility of orientation" are more likely to be promoted. 

There is some indication, therefore, that differences in organiza- 
tional values will be reflected in evaluation studies. Furthermore, as 
personnel in educational organizations shift levels within a hierarchy, 
their values with respect to evaluation may significantly shift. Thus, 
judgments of the quality, utility, and relevance of particular evalua- 
tion activities may vary considerably as one surveys across educational 
research and development: agencies, state education agencies, university- 
based evaluation centers, and local school districts, as a result of 


differences in organizational values. 


Values as Demand Characteristics 

Many values operate directly and visibly on evaluation study 
decisions, clearly suggesting which procedures are most desirable. 
Other more tacit values exert a subele dnfiuenee on the formation of 
‘evaluation studies. In many respects these more subtle values in the 
evaluation context can be thought of as "demand characteristics" of 
the @valuation enterprise, in much the same respect as Orne (1972) 
discusses the demand characteristics of the experimental setting. Orne 
defines demand characteristics fs the sum total of the cues which suggest 
to the experimental subject what behaviors are expected or desired. 


It is perhaps best to think of the perceived 
demand characteristics as a contextual variable 

in the experiment situation... It should be-clear 
that demand characteristics cannot be eliminated 
from experiments; all experiments will have 

demand characteristics, and these will always 

have some effect. (p. 779) 


Similarly, various values within the context of an evaluation act 
as strong demand characteristics, cueing the evaluator to expected or 
desired behaviors. These values may have little direct relation to the 
actual product or program being evaluated, but nonetheless have a 
considerable impact on the definition of evaluation and the activities : 5 
pursued by evaluation personnel. For example, a program long 


beleaguered by external criticism and lack of support might desire 
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that a newly hired internal evaluator adopt a highly positive and 


supportive stance toward the program. The evaluator, may be expected 

to evidence his or her commitment to the welfare of the program before 
even the mildest criticism will be accepted by the program staff. If the 
evaluator chooses the role of providing credible and helpful feedback 

to the program, then his or her interpersonal behaviors, reporting 
strategies, and perhaps even data collection procedures will be signifi- 
cantly affected by the program expectation, that he or she appear highly 
supportive. In another situation the program administrator might 
surreptitiously use evaluation activities to fulfill a management 
function of keeping program development efforts on a strict timeline. 
Such a demand characteristic might influence the evaluator to engage 
more heavily in progress monitoring, administrative feedback, and 
program planning than might. otherwise be the case. Subtle values, 
operating as .cues to Auuiena behavior (demand characteristics) can 
therefore determine which activities are finally selected as appropriate 


in a particular evaluation study. 


An Illustration: Evaluation Purposes 


Before leaving the topic of how contextual values influence the 


conduct of evaluation studies, consider one final illustration. This 


example demonstrates how the purpose of an evaluation study can be 
portrayed as the result of. several contextually influenced value / 
judgments: specifically, judgments about. 

e what to evaluate (the focus of the evaluation) , ‘ 


e who is to benefit from the evaluation (the audience of 
the evaluation), and 


e what the evaluation is to be used for (the evaluation use). 

Several years ago Scriven (1967) discussed the distinetion between 
the goals and roles of evaluation. He defined the goal of evaluation 
to be the gathering and combining of performance data with a weighted 
set of goal scales to yield either comparative or numerical ratings of 
worth. 


But. the role which evaluation has ina particular 

educational context may be enormously various; it 

may form part of a teacher training activity, of So 
the process of curriculum development, of a field 

experiment connected with the improvement of 

learning theory. -(pp. 40-41) 
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The term "purpose" is used here, for, although it is meant to be 


synonymous with Scriven's definition of "role," it does agpear to be 
a more commonly used term than "role" in-much of the evaluation 
literature. : 

Many purposes for evaluation have been proffered: evaluation for 
program planning, program improvement, RW progran justification (Brophy 
et al., 1974), evaluation to determine what Bredkdunes- dre effective 
with what kinds of students and under what circumstances (Hastings, 
1966; Rutherford, 1971), evaluation to aid in development by assisting 
with administrative decisions (Stufflebeam et al., 1971), evaluation to 
aid in understanding empirical relationships in order to enable 
revisions (Cronbach, 1963), evaluation to help developers be accountable 
to their many publics (Gooler and Grotelueschen, 1971), evaluation to 
provide for public accountability (Merriman, 1971), and evaluation to 
fulfill obligatory requirements (House, 1973a). These different 
purposes for evaluation result in different designs, methodologies, 
and evaluation approaches being sea.. 

Hemphill (1969), for example, states that the worth of an 
evaluation study depends upon its contribution to a rational decision- 
“making process in situations where it is;necessary to estimate the 
probability of desirable, but uncertain, outcomes of action chosen 
from several alternatives. This argument presumes that the primary 
purpose of the evaluation is to serve decision-making. However, 
contributions to decision-making would not be a major criterion for 
an evaluation designed to serve public accountability or to provide a 
fuller understanding of empirical relationships. 

Evaluation work is initiated presumably because there is an 
interest in uSing evaluative activities to produce some desired 
outcome, whether an information product or a situational change. In 
some cases the outcomes of evaluation work result as much from the 
evaluative process itself, and its reactive nature, as from any 
informational predict resulting from evaluation data. An evaluation 
May be put to use in a variety of ways. BvalNaLions can be used to 
influence program planning, control, refinement (doing a better job 
in the same direction), revision (choosing new directions), 


continuation, justification, validation, monitoring, accountability, 
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communication, or understanding. Which of these uses is finally 
selected in an evaluation study is the result of a series of value 
judgments based on the dominance of various contextually relevant 
values. 

The values of audience groups also influence the form of evaluation 
studies. There are at least five primary audiences in most educational 
evaluation studies: the program producer/developer, funder, evaluator, 
target group/user, and purchaser/consumer. “Other related parties which 
also may be relevant audiences include teachers, school administrators, 
school professional staff (coaches, librarians, counselors), state 
agencies, federal agencies, community groups, society in general, 
special interest or ad hoc groups, academic content experts, universities, 
parents, and researchers. Since the evaluation study is to benefit one 
or more of these groups, their information needs, interests, and basic 
values must be considered throughout the conduct of the evaluation. 

Finally, the choice of evaluation focus, that is the specific 
program components to be evaluated, is usually the result of basic 
value judgments of what constitute the most significant elements of the 
program. Although most studies focus on instructional outcomes, other 
foci such as managerial structure, dissemination capabilities and 
instructional content are also possible. 

By portraying the purpose of an evaluation study as a combination 
of focus, audience, and evaluation use, as displayed in Table 1, the 
ways in which basic value positions influence the definition of a 
particular study become more apparent. Oi 

Note the similarity between the last two purposes listea in 
Table 1: 


e An evaluation of instructional outcomes for the benefit 


of researchers for use in understanding how to better 
teach the content area. € 


e An evaluation of instructional outcomes for the benefit 


of the public for use in increasing the awareness of 
program activities. 
& 


am 


Evaluation 


Table 1: 


(FOCUS ) 


management style 


budgetary affairs 


instructional 
content 


implementation 
strategies 


implementation 
strategies 


instructional 
outcomes 


instructional 
outcomes 


Benefit of 


Alternative Purposes for an Evaluation Study 


(AUDIENCE) ®, 


the funding agency 


o 


the public 


the program 
development staff 


the program 
development staff 


product 
purchasers 


the public 


(EVALUATION USE) 


monitoring the 
program 


assessing 
accountability 


revising the 
materials 


planning 
dissemination™ 
activities 

* 


making purchas- 
ing decisions 


understanding 
how to better 
teach the con- 
tent area 


gS 
CO 


increasing the 
awareness of 
program activity 


Both focus on instructional outcomes, but for different audiences who 


make different uses of the evaluation (generalizable understanding 
versus general awareness). Since these audience groups have different 
interests to be served by an evaluation, different types of evaluation 
Studies will be necessary to respond to their respective needs. 
Selecting one of these groups as a primary audience entails a value 
judgment, and once a selection is made, that group's values become more 
influential in the conduct of the evaluation work. 

Evaluations seldom, if ever, have a single purpose. Different 
audiences arg-usually interested in different components of a program 
for different reasons. Furthermore, it is not necessary that evaluators 
consciously consider purposes in conducting evaluation studies (cf. 
March, 1972). However, by considering an evaluation's purpose and the 
types and sources of values relevant to various purposes, as illustrated 
in Table 1, one can more readily compréhend the impact of social values 


on evaluation activities. 


THE EVALUATOR AS A SOURCE OF VALUES 


Values which influence evaluation studies not only arise from the 
contexts within which such studies are conducted, they gre also inserted 
into the evaluation enterprise by evaluators themselves. Such values 
are evident in evaluation terminology, evaluation models, and in the 


personal preferences of individual evaluators. 


Values Hidden in Terminology 


Values that impinge on the conduct of evaluation work can be seen 
in the terminology and paradigms used in evaluation studies. Since 
evaluation is, at least partly, a scientific activity, a brief look at 
values in science is pertinent. Rudner (1973) argues that scientists 
cannot avoid making value judgments in their work; the scientific 
method intrinsically requires the making of value judgments and value 
decisions. Gunnar Myrdal (1969) has provided an excellent discussion 


of how subjectivity and bias eriter into all social research. 
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This implicit belief in the existence of a body of 
scientific knowledge acquired independently of all 
valuations I soon. found to be naive empiricism. 
(p. 9) 

. 


Myrdal says that scientific work requires questions which are expressions 
of our interest in the world and are, therefore, valuations. Values are 
involved as one observes facts, carries on theoretical analyses, and 
makes inferences. 

Myrdal notes that social scientists are apt to conceal conflicts 
betweén valuations by stating their positions as simply logical 
inferences from observed facts. Since they suppress valuations as 
valuations and give only "reasons," their perception of reality easily 
becomes distorted or biased. Scientists (and evaluators) cannot avoid 
bias by stopping short of practical or political conclusions, since 
such conclusions are sure to be drawn anyway , not as inferences from 
explicit value premises, but as supposedly evident from the nature of 
things, as part of the objective data. 


Biases are thus not confined to the practical and 
political conclusions drawn from research. They 
are much more déeply seated than that. They are 
the unfortunate results of concealed valuations 
that insinuate themselves into research at all 
stages, from its planning to its final presentation. 
As a result of their concealment, they are not \ 
properly sorted out and can thus be kept undefined 
and vague. (Myrdal, 1969, p. 52) 


Myrdal argues that the only way scientific inquiry can be objective is 
for value premises to, be made explicit. 

According to Myrdal, "terminological escapism" is no solution to 
the problem of values in research. He suggests that there is nothing 
wrong per se with value-loaded concepts if they are clearly defined in 
terms of explicitly stated value premises. If they are not so defined, 
and the implied valuations are therefore concealed, they provide entry 
for bias. Inventing new terms is no solution since new terms only give 
a false sense of security and serve to deceive the public. Valuations 
Must be made’explicit. Research can be made objective by allowing 
others to challenge scientific investigations on the basis of their 
explicit value premises or to reconduct the studies, substituting 


different value premises. 
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Others, like Passmorg (1973), have addressed thd 
ik 


f 


oroblem of 


terminological bias “&rgjuing that scientific-soundigg words like 
"efficiency" often disguise value judgments, and that seemingly factual 
statements often imply value judgments or even formulation of social 
policies. He maintains that scientists sinetined eoress to be giving 
merely technical advice when they are actually assuming certain value 
positions and espousing particular policy decisigns. Passmore argues 
that a social science must be aware of the value/ judgments in its work 
Hl 


‘and that , i 


a positive social science must be ee in the 
sense that it is not social advocacy’fin disguise, 
but not in the sense that it has nothing »ito say 
about values. (p. 523) a : 


Hooper (1968) reminds us that even *commn virtues" such as clear 
thinking, precise expression, unambiguous language, and being certain, 
are still only value statements or value preferences. Any virtue or 
judgment criterion is only a value preference; there may be competing 
value positions which are equally reputable. 

Hidden values in neutral-sounding terminology plague evaluation as 
well as other scientifically based enterprises. In reviewing the Phi 
Delta Kappa report on evaluation, Scriven (1972) objects to the report's 
"persuasive definition" of evaluation. He points to the authors' value 
judgments in defining evaluation to include administrative theory, needs 
assessment, monitoring, bookkeeping, and so on. He argues that such 
“activities are not direct derivatives of evaluation when defined as 


neutrally as possible. 


Value-Laden Evaluation Models 

Just as implied value judgments are hidden in the terms used by 
evaluators, so are values evident in evaluators' conceptual tools. 
Values are inherent in the paradigms used in science, and thas sees 
to be especially true with evaluation models. 

Individuals writing on the nature and use of models in science 
(e.g., Black, 1973; Kaplan, 1964) have noted several uses made of the 
term. Black, for example, discusses how the term "model" has been used 


in science to denote: 


scale models, 
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analogue models in which there is an isomorphic 
relationship between the model and the original 
entity, alc 


models which are types or designs of something 
worthy of imitation, and : ; 
theoretical models in which one uses an area 
better understood in order to solve a particular 
problem. 


The term "models" as typically used in educational evaluation appears to 
refer to the third definition above; i.e., evaluation models are designs 
of exemplary processes or procedures. 

Evaluation models are not theoretical models since they lack: 
explanatory and predictive features. They are more properly considered 
as exemplars of desirable or commendable operating procedures, and are 
best viewed as alternative, descriptive conceptualizations of evaluation 
processes. Black describes such a use of models as a "convenience of 
expoSition...which may also help us to notice what otherwise may be 
overlooked, to shift the relative emphasis attached to details--in short, 
to see nefé'connections" (p. 496). Evaluation models, therefore, repre- 
sent competing conceptualizations of appropriate evaluation procedures; 
each model is based on differing assumptions and values concerning the 
evaluation process itself. - =F 
4 


Evaluation models are not always accepted as value-based exemplars 


suitable for heuristic use, but are sometimes viewed as objective 


' 


' 


methodological procedures (Smith and Murray, 1975). Obvious problems 
arise when enthusiasts for a particular model lose sight of the value- 
based nature of their paradigm and approach the use of the model asia 
"scientific" and value-free activity. 

House (1973d) notes that 


eas with other technologies, the technology of 
evaluation has shaped the minds of those who use 
it, the evaluators. Evaluation problems are 
automatically reduced to testing problems and to 
problems of measuring objectives. If the evaluator 
chances to’ cast the problem differently, he finds 
that this new formulation is perceived as 
illegitimate, uneconomical, and unacceptable. 
Even slight deviations in instrumentation are 
controlled by the norms of "reliability" and” 
"validity." Powerful institutional forms ‘have 
grown up Mound the technology as devices for 


at ; 


maintaining the status qua...The governmental 
agencies accept only "hard" data. (p. 21) 


House is arguing against the underlying values aes currently 
prevalent approach to evaluation. He asserts that other social values 
should be dominant and should direct* evaluation work: that means-ends 
paradigms should be distrusted and "more trust fbe] placed in the 
evaluator to respond as-a human being, as a ariticat intellect, less on 
tachniaue, He tehouLd look) for natural patterns rather than trying to 
control" (p. 23). House proposes that evaluation ought to be opposed to 
the following values: 


componentiality: reality can be easily separated 
into neat components, 4 


multirelationality: an individual can be fragmented 
into all the relationsfiips he must maintain in the 
society, °* 


C* 


makeability: problems can always be solved, and 

progressivity: it is desirable to maximize the 

benefits of any action. . 
Whether or not one agrees with House, it is clear that value positions 
Play a fundamental role in the selection of evaluative approaches. 

a . \ 
An Evaluator's Personal Values 
There are often considerable differences between the values of an 

evaluator and the values of the individuals he works with and for. 
These value differences, whether made explicit or not, can influence the 
nature of the evaluation work as it unfolds. Several authors have 
_stressed the need for greater attention to the personal values of the 
: participants in evaluation activities. 
House (1973c), in discussing the value differences between evalua- 


\“ 


there is a natural antipathy between’change and 
evaluation, oftén unrecogniged by the evaluator \ 

and client alike. Changing ,Something requires a . 
faith, a belief in the new program beyond any data. 

To the client, evaluation means a confirmation of 

what he knows to be true. Evaluating something 

means being skeptical, suspending belief. But the 
evaluator has a faith of his own--that the client 

will change as a result of the evaluation. (pp. 256-257) 


a 


tors and clients, says, 
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Similarly, Weiss (1973) cautions evaluators to remember that 


administrators are frequently as much or more concerned with building 
and maintaining an organization as they are with seeing that program 
goals are met. She argues that evaluators must realize that maximizing 
program effectiveness is only one of the many values that enter into 
the politically oriented decisions made by administrators, and into 


which evaluation feeds information. 


\ Accomplishing the goals for which the program was 
set up is not unimportant, but it is not the only, 
the largest, or usually-the most immediate of the 
concerns on the administrator's docket. (p. 38) 
Weiss notes that whereas the evaluator wants to study the effects of a 
Stable and specifiable stimulus, the program administrator wishes to 
make the best possible adaptations to changing contextual conditions. 
Because ¢vdiuation is a reformist movement of social science 
evaluators who are often more iiheval. in orientation than the programs 
they study, the evaluators' perspectives inevitably affect the nature 
of their studies. Serious value conflicts surface when the evaluator 
follows Scriven's (1972) advice to examine and critique the client's 
objectives and values. Scriven claims that it is necessary that the i 
evaluator act as a conscience as well as a consultant to the effort 
being evaluated. But Scriven warns that the evaluator must also reveal 
his own values. 
Berlak (1970) stresses the critical necessity of evaluators' making 
their own values clear because of their frequent role as moral judges. 


Though we cannot suryive without experts, they can 
also do us in. I am suspicious of immodest experts; 
experts by virtue.of their expertise certainly do J 
not possess superior moral values. Critics of 
democracy have long pointed out that the hazard of 

a democratic system is that the people may not choose 
the wisest men to govern. This unhappy consequence 
disturbs me less than the expert who presumes he 
knows best what is good for society. (p. 277) 


An evaluator's values can affect many aspects of an evaluation 
study including its design, data collection strategy, data analysis, and 
reporting. Barber (1973) has enumerated nine different effects, such 
as loose protocols, misrecording errors, and unintentional expectancy, ~~ 
which may result from the values of individuals conducting research 
studies. Barber's riectey of the research on experimenter expectancy 
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effects illustrates that this problem is more pervasive than is 


generally thought. Finally, Werts and Linn (1969) have shown that 
even the choice of particular regression analysis procedures can be 
influenced by the hypothesis one wishes to support; that is, the value 
position of the evaluator. These potential bias effects are always 
present in evaluation studies, and provide mechanisms by which the 


personal values of évaluators can affect evaluation results. 


An Illustration: Evaluator Roles 

As one final illustration of the influence of values in evaluation 
work, consider how the xele of an evaluator in a particular study can be 
portrayed as the result of several value judgments concerning 


e what is the purpose of the study (purpose), 


e to what audience group does the evaluator owe prime 
allegiance (allegiance group), and 
e what stance is the evaluator taking in relation to the ; 


program's worth (posture) . 

Of the audiences previously discussed, those of interest here are 
the audience groups to whom the evaluator feels allegiance or in whose 
interest he feels his work is being conducted. For instance, an 
evaluator may feel that his prime responsibility is to aid the program 
Gey SLODMene staff, or to provide a monitoring function to the funding 
agency, or to protect the interests of a particular consumer group. One, 
of these audiences thus becomes the evaluator' s allegiance group. (There 
has been discussion as to whether evaluators ought to be employed 
internally or externally to the program being evaluated. The 
considerations of pnternality/external tty usually concern the degree 
of cooptation and allegiance reflected in the evaluator's work. Since 
funding and employer relationships do not negessarily dictate allegiance, 
it seems more useful to consider allegiance directly rather than 
internal or alae amisie is, arrangements. ) ‘ ” 

The posture of an evaluator refers to whether or not the evaluator 
is attempting to take a neytral, positive, or negative stance toward 
the program being evaluated. Useful evaluations may be conducted by - 
evaluators who have either positive or negative attitudes towards a : 
program, provided such biases are made public, taken into consideration 


as one ubiews the evaluation work, and counterbalanced by views from 
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alternative perspectives. (Such a view is consistent with Myrdal's 


(1969) view, which argues that there is no such thing as objectivity 
except insofar as it refers to particular value biases being made 
explicit and public.) 

Using purpose, allegiance group, and posture, one can easily 
enumerate various stereotypic evaluator roles as indicated by the 


examples below: 


Evaluator Role Purpose Allegiance Group Posture 


researcher under- content neutral 
standing specialists 
‘auditor monitor funder neutral 
. co-worker program program staff positive 
refinement 
consumer accounta- potential users negative 
protector bility 
certifier validation society neutral 


These examples are meant to be illustrative only. Whether a consumer 
protector role should be played from a neutral or negative stance is not 
the point here. What is of interest is that an evaluator's role can be 
“seen as the result of a series of value judgments about which group to 
serve, what stance to take, and what the primary purpose of the evalua- 


tion ought to be. 


VALUES CLARIFICATION IN EDUCATIONAL EVALUATION 
Benefits and Problems of Values . 

Values certainly pervade every aspect of evaluation work and arise 
from a multitude of sources, from social movements to the evaluator's 
personal opinions. It is fortunate that since such values are 
inescapable, they are also beneficial. Value positions help to define 
and delineate an evaluation problem and, to a large extent, to determine 
its practical significance for particular audiences. Furthermore, 
although evaluation terminology and methods are value laden, they are 
essential tools. There is obvious utility in the simplifying nature 
of models. Models, and especially their underlying values, help to 
reduce complexity and ptovide decision-making mechanisms by which an 


overwhelming number of alternatives can be managed in conducting an 
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evaluation study. The value dimensions of models help delineate what one 
will look at and how it will be described. Such models serve a useful 
‘function of simplification. 

Difficulties do arise, however, in evaluation studies where the 
values influencing the results are not clear and explicit, and the 
evaluative research is therefore portrayed or viewed as something other 
than it actually is. In natty studies, pluralistic values, and the 
resultant countervailing forces, complicate the eva tdakion work. Various 
audience groups advocate different objectives as the "true" purpose of 

the eae study and pressure evaluators into "hatchetman" or “white- 


wash" \roles. Other individuals attempt to shift the initial focus of 


v 
the evaluation study from, for example, student outcomes to personnel 

adequagy. Under such circumstances, evaluators are often hard pressed 
to idektify the various interest groups and values being promoted,. and 


to undexystand how such values may be affecting the direction of the 


evaluati ork. They need a means of dealing with such influential 


values. 


ao 


Ways to Clarify Values 

Although telativedy little has been done to help the practitioner 
cope with competing value systems, four approaches to dealing with values 
can be. tentatively identified in the evaluation literature. All the 
approaches are primarily designed to clarify different value positions 

eee on the presumption that it is the lack of awareness of value positions, 

rather than the eaine positions themselves, that presents the greatest 
difficulty in evaluative inquiry. . 

The first approach simply suggests that all relevant value positions 
need to be identified snd, stated publicly. ‘House (1973a), for example, 
echoes Myrdal's (1969) argument fer values explication and urges that 


making evaluations explicit demonstrates the 
evaluator's awarehess of them, forces him to 
‘account for them, &nd exposes them for what they 
are. Ideally, one woyld use alternative sets 
. of values to judge a program. (House, 1973a, 
RB: 131) 
House concedes that there are seldom sufficient resources available to 
° use multiple value perspectives, and so ont "is usually forced to choose 
‘a particular set of valuations and to utilize resources eaea 
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that perspective. But, he argues, these steps should be taken publicly 
so that the valuations chosen are neither hidden nor arbitrary, and so 
that they can be judged for their relevance and significance by the 
audiences involved. This approach emphasizes making public all hidden 
valuations. 

The second approach emphasizes the need to clarify the evaluator's 
role in the evaluative process. Rex lak (1970) suggests three different 
positions an evaluator may take with regard to making value judgments 
about the worth of a program being assessed: the evaluator may (a) only 
describe the program, (b) recommend judgment criteria, or (c) render an 
actual judgment of worth. The evaluator may thus choose, to some degree, 
how influential his or her values will be in the final assessment of 
worth. Of course, if the evaluator chooses to de-emphasize his or her 
personal values, the questions of whose values are being influential 
remains open. 

The third approach suggests explicitly incorporating opposed values 
into evaluation studies by conducting comparative analyses. Such 
studies might employ tests of critical competitors (Scriven, 1974) and 
political benefit analysis. 


Just as economic cost-benefit analysis added the 
vital dimension of cost to the analysis of outcomes, 
political-benefit analysis might help to resolve 
questions about political benefits and EoEedOn= 
opportunities. Weiss», 1973, p. 40) 


The fourth approach patleets attempts to actively search out 
conflicting value positions to insure an appreciation of the full 
range of potentially influential values. Asking certain groups to be 
public about their values is part of this process, but to keep these 
self-disclosures in perspective, other groups who advocate eouiter 
positions are likewise sought and heard. The interplay of multiple 
values provides a stronger means of values regulation than only, making 
explicit statements of one position. In fact, publically including 
multiple value. positions may ease the evaluator's moral burden by 
shifting his or her position from that of judge to arbitrator (Brickell, 
1976). Examples of this approach include adversary procedures (Levine, 
1974) and portrayal evaluations (Stake, 1975). This fourth approach 


emphasizes the use of multiple value perspectives in order to achieve a 


" ' 
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fuller understanding of the educational program, while the. third 
approach emphasizes procedures which test the dominance of one value 
position over another. 

These various approaches to dealing with values in evaluation work 
provide important first steps, but much work remains to be done. 
Currently the evaluation profession lacks (a) .a suitable theory of 
value related to evaluative inquiry, (b) procedural guidelines for 
dealing with values in applied settings, and (c) an empirical base 
indicating the impact and role of values in evaluative work. Having 
identified some of the important sources of values in evaluative 


inquiry, perhaps work can proceed with a little clearer map of the 
A 


terrain. 
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